Multimodal ocular biometric system and methods

ABSTRACT

Biometric systems capture and combine biometric information from more than one modality, employing digital processing algorithms to process and evaluate captured images having data for a biometric characteristic. Such digital algorithms may include a pupil segmentation algorithm for determining a pupil image in the captured image, an iris segmentation algorithm for determining an iris image in the captured image, an eyelid/eyelash segmentation algorithm for determining an eyelid/eyelash image in the captured image, and an algorithm for measuring the focus on the iris. Some embodiments employ an auto-capture process which employs such algorithms, in part; to evaluate captured images and obtain the best possible images for biometric identification.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a Divisional Application of U.S. application Ser.No. 11/898,190 filed Sep. 10, 2007, now U.S. Pat. No. 8,170,293 whichclaims priority to U.S. Provisional Application No. 60/844,659 filedSep. 15, 2006, each of which are hereby incorporated by reference hereinin their entireties.

BACKGROUND OF INVENTION

1. Field of Invention

The present invention relates generally to instruments for biometricidentification, and more particularly, to a multimodal ocular imagingsystem used for biometric identification and methods for processingimage data captured by the multimodal ocular imaging system.

2. Description of the Related Art

Due to the unique character of each individual's retina or iris, varioussystems attempt to use either the retina or the iris for biometricidentification. Commercially available ocular imaging systems used forbiometric identification generally use a single biometric modality.These imaging systems process images of the iris or the retina from onlyone of two eyes of a subject. None of these conventional systemsprocesses images of both the iris and the retina in combination.Moreover, these systems do not process images from the iris and/or theretina from both eyes.

Conventional single-eye iris imaging systems suffer from severaldisadvantages. In particular, such systems may suffer from frequentfailure to acquire an image, i.e. a high fail-to-acquire (FTA). Theeffectiveness of these iris imaging systems is often limited byocclusions caused by eyelids and eyelashes, lighting issues (controlledor uncontrolled), focus problems, pupil size variation (betweendifferent persons or with the same person), non-linear iris fiberdistortion caused by expansion or contraction of the pupil, and rotationand skew of the head or eye. Such systems are also susceptible tospoofing. Moreover, auto focus functions of conventional iris-onlysystems are affected by scratches in eyeglasses or the reflections fromeyeglasses. In fact, ANSI standards require enrollment to be withouteyeglasses. Additionally, contact lenses can cause iris outer boundarysegmentation problems. Moreover, colored contact lenses can result inspoofing.

Conventional single-eye retina imaging systems also have severaldisadvantages. For instance, problems with such retina imaging systemsoccur when visible light used for illumination blinds or distracts theuser, when the user is not properly aligned with the image capturedevice, or when poor areas of the retina are chosen for imaging.Moreover, conventional retina-only systems are also negatively affectedby focus problems as well as rotation and skew of the head or eye.

In addition, as a further disadvantage, the conventional imaging systemsabove process captured image data according to exhaustive edgedetection, computationally expensive circle finding techniques, andother algorithms that are less appropriate for real time use and use onconventional processing devices.

While the iris systems described previously only process an iris imagefrom only one of two eyes, there are other existing devices that acquireiris images from both eyes. However, such systems suffer fromsignificant disadvantages. For example, these existing devices require asubject to walk up to a substantially stationary and device and look ata half mirror to position his eyes properly for image capture.Disadvantageously, this approach requires the subject to positionhimself so that his eyes are at the “right” height for image capture, oralternatively, the acquisition device must repositioned to accommodatethe height of the subject, which may vary from 4 feet to 7 feet.

SUMMARY OF THE INVENTION

Considering the disadvantages of the single modal systems describedpreviously, a need has been identified for a multimodal ocular biometricsystem that addresses these disadvantages by capturing and combiningbiometric information from more than one modality. In particular,embodiments of the present invention provide a multimodal ocularbiometric system that captures and processes images of both the iris andthe retina, from which data'can be determined for biometricidentification.

Further embodiments provide a multimodal ocular system that captures andprocesses images of the iris and/or the retina from both eyes of asubject. For example, one embodiment may provide a dual-iris multimodalocular system that processes images of the iris from both eyes of asubject. In contrast to some devices described previously, embodimentsof the present invention may have a size and shape that is convenientfor operation of the embodiments. Particular embodiments may havebinocular-like shape that permits a user, regardless of height, to bringthe device to the user's face and correctly correct position the user'seyes for image capture.

Embodiments of the present invention employ algorithms to control thecapture of biometric information from more than one modality and toprocess the captured images. For example, the dual-iris embodimentsdescribed immediately above may employ digital processing algorithms toprocess and evaluate iris image data. Such digital algorithms mayinclude a pupil segmentation algorithm for determining a pupil image inthe captured image, an iris segmentation algorithm for determining aniris image in the captured image, an eyelid/eyelash segmentationalgorithm for determining an eyelid/eyelash image in the captured image,and an algorithm for measuring the focus on the iris. Some embodimentsemploy an auto-capture process which employs such algorithms, in part,to evaluate captured images and obtain the best possible images forbiometric identification. The digital algorithms may be implemented on aprocessing device, which executes programmed instructions correspondingto the digital algorithms.

In one embodiment, pupil segmentation and iris segmentation are achievedthrough a sparse point method (SPM) algorithm. In another embodiment,the eyelid/eyelash segmentation employs iris intensity modeling fromregions free of eyelid/eyelash occlusion as a basis for determiningwhether other regions of an eye image correspond to eyelid/eyelashocclusion. In yet another embodiment, the eyelid/eyelash segmentationalgorithm employs iris texture analysis using a bank of log-Gaborfilters to generate a texture representation based on the phasecongruency feature-space. In a further embodiment, an algorithm formeasuring the focus on the iris employs a gradient technique across theiris/pupil boundary. In yet a further embodiment, an iris focusmeasurement employs the lighting reflection from image capture.

Still other aspects, features, and advantages of the present inventionare readily apparent from the following detailed description, byillustrating a number of exemplary embodiments and implementations,including the best mode contemplated for carrying out the presentinvention. The present invention is also capable of other and differentembodiments, and its several details can be modified in variousrespects, all without departing from the spirit and scope of the presentinvention. Accordingly, the drawings and descriptions are to be regardedas illustrative in nature, and not as restrictive.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 illustrates an embodiment of the present invention with aquadruple-sensor, two-eye simultaneous configuration.

FIG. 2A illustrates a retina auto-focus technique employed by anembodiment of the present invention.

FIG. 2B illustrates another retina auto-focus technique employed by anembodiment of the present invention.

FIG. 2C illustrates yet another retina auto-focus technique employed byan embodiment of the present invention.

FIG. 3 illustrates an embodiment of the present invention with adual-sensor, two-eye flippable configuration.

FIG. 4 illustrates an embodiment of the present invention with atriple-sensor, two-eye sequential configuration.

FIG. 5 illustrates an embodiment of the present invention with asingle-sensor, two-eye sequential configuration.

FIG. 6A illustrates an external view of an embodiment of the presentinvention that captures iris images from both eyes.

FIG. 6B illustrates another external view of the embodiment of FIG. 6A.

FIG. 6C illustrates the use of corrective eyewear with the embodiment ofFIG. 6A.

FIG. 6D illustrates an internal view of the embodiment of FIG. 6A.

FIG. 7 illustrates an exemplary fixation scheme as seen by the user.

FIG. 8A illustrates another exemplary fixation scheme, as seen by theuser when the user is misaligned along the X-, Y-, and Z-axes.

FIG. 8B illustrates another exemplary fixation scheme, as seen by theuser, when the user is misaligned along the Z-axis.

FIG. 8C illustrates another exemplary fixation scheme, as seen by theuser when the user is aligned along the X-, Y-, and Z-axes.

FIG. 9 illustrates an exemplary scheme for entering a personalidentification number by pupil tracking.

FIG. 10 illustrates digital processing algorithms that may be includedin an embodiment of the present invention.

FIG. 11 illustrates an exemplary digital processing algorithm for pupilsegmentation and iris segmentation which may be employed by embodimentsof the present invention.

FIG. 12 illustrates aspects of applying the exemplary digital processingalgorithm of FIG. 11 to a captured eye image.

FIG. 13 illustrates further steps employed by the exemplary digitalprocessing algorithm of FIG. 11 corresponding to eyelid/eyelidsegmentation.

FIG. 14 illustrates an exemplary digital processing algorithm foreyelid/eyelash segmentation which may be employed by embodiments of thepresent invention.

FIG. 15A illustrates an annular iris region calculated for captured eyeimage, which is processed by the exemplary digital processing algorithmof FIG. 14.

FIG. 15B illustrates a rectangular image which results from unwrappingthe annular iris region of FIG. 15A according to the exemplary digitalprocessing algorithm of FIG. 14.

FIG. 15C illustrates exemplary results of coarse, or fast, detection todetermine a coverage measure corresponding to eyelid/eyelash occlusion,as determined by the exemplary digital processing algorithm of FIG. 14.

FIG. 15D illustrates exemplary results of pixel-wise mask generation todetermine a coverage measure corresponding to eyelid/eyelash occlusion,as determined by the exemplary digital processing algorithm of FIG. 14.

FIG. 16 illustrates an example of the absolute difference in percentagecoverage measure between ground truth and fast, or coarse, detection(marked as “Fast Detect”) and between ground truth and pixel-wise maskgeneration (marked as “Mask”).

FIG. 17 illustrates another exemplary digital processing algorithm foreyelid/eyelash segmentation which may be employed by embodiments of thepresent invention.

FIG. 18 illustrates further steps employed by the exemplary digitalprocessing algorithm of FIG. 17.

FIG. 19A illustrates an unwrapped iris images for application of anexemplary digital processing algorithm for eyelid/eyelash segmentation.

FIG. 19B illustrates the mask corresponding to the unwrapped iris imageof 19A after application of an exemplary digital processing algorithmfor eyelid/eyelash segmentation.

FIG. 19C illustrates an unwrapped iris images for application of anexemplary digital processing algorithm for eyelid/eyelash segmentation.

FIG. 19D illustrates the mask corresponding to the unwrapped iris imageof 19C after application of an exemplary digital processing algorithmfor eyelid/eyelash segmentation.

FIG. 19E illustrates an unwrapped iris images for application of anexemplary digital processing algorithm for eyelid/eyelash segmentation.

FIG. 19F illustrates the mask corresponding to the unwrapped iris imageof 19E after application of an exemplary digital processing algorithmfor eyelid/eyelash segmentation.

FIG. 19G illustrates an unwrapped iris images for application of anexemplary digital processing algorithm for eyelid/eyelash segmentation.

FIG. 19H illustrates the mask corresponding to the unwrapped iris imageof 19G after application of an exemplary digital processing algorithmfor eyelid/eyelash segmentation.

FIG. 20A illustrates an exemplary digital processing algorithm for irisfocus measurement, which employs a gradient technique across theiris/pupil boundary.

FIG. 20B illustrates another exemplary digital processing algorithm foriris focus measurement, which employs a gradient technique across theiris/pupil boundary.

FIG. 21 illustrates an exemplary approach for capturing an iris image,which may be employed by embodiments of the present invention.

FIG. 22 illustrates further aspects of the exemplary approach of FIG. 21for capturing an iris image.

DETAILED DESCRIPTION

Embodiments of the present invention provide a multimodal ocularbiometric system that captures and processes images of both the iris andthe retina, from which data can be determined for biometricidentification. Further embodiments provide a multimodal ocular systemthat captures and processes images of the iris and/or the retina fromboth eyes of a subject. Biometrics based on data provided by theseembodiments are more accurate and robust than using biometrics thatinclude data from only the iris or only the retina from a single eye.

Advantageously, the iris and retina present biometric features that areboth independent and strongly coupled. They are independent in that theyare extracted from different biological structures. On the other hand,the iris and retina biometric features are strongly coupled becausethere is a fixed geometric relationship between the iris and the retina.Specifically, the position and orientation of the eye is reflectedsimultaneously in both the iris and the retina. Further, the biometricfeatures of the iris and the retina are on the same scale. The strongcoupling between the biometric features of the iris and the retina notonly facilitates the simultaneous capture of these biometric features,but allows these features to be cross-referenced or combined in a commonfeature space that preserves the geometric relationship between the irisand retina. In addition, the use of an iris system complements the useof a retina system. For instance, small pupils are generally anadvantage for iris systems while large pupils are generally an advantagefor retina systems.

Accordingly, embodiments of the present invention employ variousconfigurations of at least one imaging system that captures iris imagesand retina images. For example, FIG. 1 illustrates a two-eyesimultaneous iris/retina combination system, which employs two irisimaging systems that respectively capture iris images of the right andleft eyes, and two retinal imaging systems that respectively capture theimages of the right and left retina, all simultaneously, or at leastsubstantially simultaneously. Information from the imaging systems isused to accomplish retinal pattern recognition, iris patternrecognition, and biometric fusion. Moreover, the information from theindividual imaging systems are used in combination to establish a hostof attributes including, but not limited to, positioning, tracking,focus, and interpupillary distance. In addition, the multimodal ocularbiometric system is especially well suited for image capture of botheyes when the user is not wearing corrective eyewear. Although many ofthe features of the present invention may be described with respect tothe two-eye simultaneous iris/retina combination system shown in FIG. 1,other configurations, as described further below, can implement thesefeatures in order to combine iris and retina images for biometricidentification.

Advantageously, embodiments of the present invention may have a size andshape that is convenient for operation of the embodiments. As describedfurther below, embodiments may have binocular-like shape that permits auser, regardless of height, to bring the device to the user's face andcorrectly correct position the user's eyes for image capture.

Referring to FIG. 1, the multimodal ocular biometric system 10 includesan optical system which is symmetric for both the left eye 2 and theright eye 4. The multimodal ocular biometric system 10 includes twocamera sensors 110 to capture respective images of the iris in the rightand left eyes. The system 10 also has two camera sensors 210 to capturerespective images of the retina in the right and left eyes. As such, aniris imaging system 100 and a retina imaging system 200 are provided foreach eye. Therefore, iris and retina images can be capturedsimultaneously, or at least substantially simultaneously. Preferably,the iris imaging systems 100 and the retina imaging system 200 arehoused in a single image capture device 12, as depicted in FIG. 1.

The biometric information collected from the system 10 includes irispatterns and retina patterns, from which biometric data can beextracted. Liveness detection, which detects whether the biometricinformation comes from a living source, may also be achieved with thesystem 10. U.S. patent application Ser. No. 11/258,749, filed on Oct.26, 2005, describes a Method and System for Detecting BiometricLiveness, and is entirely incorporated herein by reference.

Furthermore, as described in more detail below, by capturing images ofboth irises simultaneously, the system 10 is able to provide biometrics,such as interpupillary distance and limbus diameter for both the rightand left eyes. Advantageously, measurements of the interpupillarydistance and limbus diameter can be used to improve database searchingduring biometric identification, because they allow reference data to bebinned and narrowed to a relevant subset of data before a search isconducted for matches based on iris codes or retinal codes. In this way,a comprehensive search of all reference data for biometric matching isnot required. For instance, limbus diameters for the general populationhave a range of about 9.5 mm to 13.5 mm. Thus, if the system 10 measuresa limbus diameter to be 10.5 mm, a subset of reference data coveringindividuals with limbus diameters in a range of 10.25-10.75 mm, ratherthan the entire database, may be searched. Compared to conducting acomprehensive search, the time to obtain a match with the reference datamay improve by up to 8 times when narrowing the data down according toranges of limbus diameter in this manner. Moreover, interpupillarydistances for the general population have a range of ±10 mm. Obtaining a±1 mm resolution would thus improve search times by up to a factor of10. As a result, narrowing the search data according to limbus diameterand the interpupillary distance may improve search times by 80 (8×10),which may be significant for very large databases. Also, throughput canbe enhanced by system memory caching based on bins for mid-sizeddatabases in multi-machine systems. Considering N interpupillarydistance bins, if N machines with N local system memories each haveenough system memory to hold the entire bin for an interpupillarydistance in the database, then database access is less likely to becomea system bottleneck.

To capture the iris and retina images, the multimodal ocular biometricsystem 10 employs both iris illumination adapted to emit photons to theiris of an eye and retina illumination adapted to emit photons to theretina of the eye. In particular, the embodiment shown in FIG. 1 employsLEDs (light emitting diodes) 120 and 220 to produce iris illuminationand retina illumination, respectively. FIG. 1 also shows that the irisand retina illumination uses separate LED's. Correspondingly, the camerasensors 110 are configured to capture iris images when the right andleft irises reflect the emitted light from the illumination source 120,and the camera sensors 210 are configured to capture retina images whenthe right and left retinas reflect the emitted light from theillumination source 220.

Alternatively, other embodiments of the present invention may employlaser diodes rather than LEDs. In these alternative embodiments, thesystem can perform laser Doppler imaging using an addressable CMOSdetector on specific regions of interest. Advantageously, this approachpermits retinal liveness testing as well as retina vessel determinationand contrast enhancement.

As depicted in FIG. 1, a controller 15 is operably connected to the irisillumination and the retina illumination, such as LED's 120 and 220. Thecontroller 15 manages the manner in which the iris illumination and theretina illumination emits photons to the irises or the retinas,respectively. As is known, the controller 15 may be a programmableprocessing device that executes software, or stored instructions. Forexample, the controller 15 may employ an external conventional computernetworked with the image capture device 12. Alternatively, a fieldprogrammable gate array (FPGA) or digital signal processor (DSP) may beemployed on board the image capture device 12. In general, the systemsdescribed herein may employ a controller, as well as other processors,that are either internal or external to the image capture devices, whichhouse the illumination and sensor systems.

The wavelengths for illumination of the iris and retina may be in thenear infrared (NIR) (700 nm to 1000 nm). Special filters or coatedoptics may be used in the optical train to select specific wavelengthsto satisfy the 700 nm to 900 nm wavelength requirements for the ANSIspecification for Iris Image Interchange Format (ANSI INCITS 379-2004),but still allow a visible color image.

Accordingly, in the exemplary embodiment illustrated in FIG. 1, the irisillumination system of the present invention may operate to illuminatejust the inner orbit of the eye. Preferably, the area of interestoutside the field of view (FOV) of the iris camera 110 is notover-illuminated, as illumination outside the FOV can cause reflectionsoff the cheek, forehead, or nose creating non-uniform illumination ofthe iris. This is especially the case for people who wear makeupcontaining TiO₂. Moreover, illuminating the area outside the FOV of thecamera is a waste of light and energy. Two arrays of LEDs 120 at awavelength of 850 nm, for example, are masked and focused on the irisand sclera in order to create this uniform illumination. Theillumination occurs at an angle of approximately ±15 degrees measuredfrom the line of sight of the user in order to minimize retro-reflectionoff the retina with the associated bright pupil corresponding to theiris image. The pupil must remain dark for image analysis and to meetANSI INCITS specification.

Light reflecting off the iris passes through a broadband antireflectioncoated optical window 330 and is imaged back through the imaging system,through a dichroic beamsplitter 130. The light then passes through aplastic or glass longpass filter 180 with a cutoff wavelength of 780 nm,for example. The longpass filter 180 prevents ambient visible light fromentering the imaging system and creating noise in the image. The lightis then focused with the iris imaging lens 190 to the image sensor 110.In a particular embodiment, the sensor 110 is a CMOS (complementarymetal-oxide semiconductor) detector with high sensitivity to NIRillumination. The CMOS detector may have square pixels, a wide angleformat, and a global shutter.

In general, the iris imaging system may have a refractive lens (a singleor a series of lenses) 190 which images the iris to a CMOS image sensor110 or, alternatively, a CCD (charge-coupled device) sensor 110. Theimage capture device 12 may also employ reflective or a combination ofrefractive and reflection optics. The imaging sensor 110 may also have aglobal shutter or a rolling shutter.

As illustrated in FIGS. 10-22, embodiments of the present invention mayemploy an iris imaging system that uses digital processing algorithms1010 to process and evaluate iris image data, which is captured, forexample, by the camera sensor 110 in the multimodal ocular biometricsystem 10. As shown in FIG. 10, the digital algorithms 1010 may includea pupil segmentation algorithm 1011 for determining a pupil image in thecaptured image, an iris segmentation algorithm 1012 for determining aniris image in the captured image, an eyelid/eyelash segmentationalgorithm 1013 for determining an eyelid/eyelash image in the capturedimage, and an algorithm 1014 for measuring the focus on the iris. Thedigital algorithms 1010 may be implemented on a processing device, whichexecutes programmed instructions corresponding to the digital algorithms1010. For example, the controller 15, as shown in FIG. 1, may beresponsible for executing instructions associated with the digitalprocessing algorithms 1010. Alternatively, a separate processing devicemay be employed to execute the digital processing algorithms 1010, but adata communications link with the controller 15 may be required toenable the controller 15 to use data calculated by the digitalprocessing algorithms 1010 as described further below.

FIG. 11 illustrates a digital processing algorithm known as sparse pointmethod (SPM), which embodiments of the present invention may employ tosegment a pupil image and an iris image from a captured eye image. FIG.12 illustrates aspects of implementing SPM to a captured eye image 1001.As shown in FIG. 12, SPM models the iris image 1002 in a captured eyeimage 1001 as an annular region 1154 with an inner boundary 1137 and anouter boundary 1153. The inner boundary 1137 is defined by a darkerpupil region 1003, while the outer boundary is defined by a lightersclera region 1004. The inner boundary 1137 of the iris imagecorresponds with the pupil boundary, so segmentation of the pupil image1003 is also a part of segmenting the iris image 1002.

In particular, SPM identifies sparse edge points in the captured eyeimage 1001 and determines how these sparse edge points relate to theinner boundary 1137 and the outer boundary 1153 of an iris image. In anintensity profile of the iris image 1002, both the inner boundary 1137and the outer iris boundary 1153 may be characterized by a light-to-darkimage intensity transition, i.e. edge, on one side and a dark-to-lightimage intensity transition on the other side. Accordingly, the sparseedge point detection uses image intensity profiles along a specifiedsearch direction in specially configured search windows. Peaks in theresulting gradient profile are then analyzed. If a determined peak isfound to satisfy a confidence threshold, then according to thealgorithm, an edge of a feature of interest has been detected. Thistechnique of edge detection is sometimes referred to as a caliper. Forgiven thresholds, the number of edges produced by each window depends onthe content of the image in that particular window region. For example,when a caliper crosses a pupil, the results include the pupil edges.FIG. 12 shows a series of rectangular search windows 1104 that may beapplied to an eye image 1001 as caliper test regions.

When sparse edge points corresponding with the iris image 1001 areidentified, an ellipse fit method may be applied to the points to deriveboth the inner boundary, or pupil boundary, 1137 and the outer irisboundary 1153 for the iris annular region 1154. Of course, a circle mayalso be used to model the iris boundaries 1137 and 1153 using thistechnique, as a circle is merely an ellipse with identical semi-axes.Alternatively, because actual boundaries found are usually not a perfectcircle or a perfect ellipse, a more general model template of theboundary, such as arc segments, may be employed to fit to the pointsfound on the boundaries. Thus, although embodiments described herein maydescribe the use of an ellipse model, or more particularly a circlemodel, it is understood that that a more general template model may beused in place of the described models.

It is understood that pupil and iris radii may vary significantly fromperson-to-person, as well as image-to-image. Moreover, in addition tothe fact that iris boundaries do not form perfect ellipses or circles,it is understood that iris locations may vary from image-to-image.However, this algorithm described herein is sufficiently robust andaccurate to accommodate such variations.

Advantageously, the use of SPM is significantly faster than techniquesthat require exhaustive edge detection and computationally expensivecircle finding techniques. As such, SPM provides a highly accuratereal-time pupil and iris segmentation algorithm, which may beconveniently implemented, for example, on a conventional personalcomputer.

As illustrated in FIG. 11, in an initial step 1101, the digitallycaptured eye image 1001 made of pixels is received. Applying SPM, step1103 identifies sparse edge points 1105 from caliper regions 1104 acrossthe entire image 1001, as shown in FIG. 12, including sparse edge pointscorresponding to a pupil image 1003. From the sparse edge points 1105,step 1107 determines regions of interest (ROI), or segments, 1109 thatare defined by a light-to-dark edge on the left and a dark-to-light edgeon the right. The data for segments 1109 include center points 1110 foreach of the segments 1109. FIG. 12 illustrates sparse edge points 1105on the left and right sides of the pupil, as well as center points 1110for the segments 1109.

In step 1111, the segments 1109 from step 1107 are grouped into clusters1113 according to the position of their computed centers 1110 and apredetermined tolerance on positional variance 1112. Accordingly, eachcluster 1113 represents an image feature that is symmetric with respectto a vertical line. Step 1111 produces any number of clusters 1113.Thus, step 1115 selects the clusters 1117, also known as coherentclusters, which correspond to a circular or circular-like feature. Theremay also be any number of these coherent clusters 1117. Therefore, instep 1119, the coherent clusters 1117 are weighted with scores based onhow closely they meet criteria 1120 that distinguish the near-circularshape of the pupil from other circular or circular-like features in theimage, such as the outer iris boundary. The scoring of step 1119 mayaccount for a number of factors, such as intensities around edge points,the number of segments in a cluster, the average segment length,proximity of segment centers to image centers, or the like. Accordingly,step 1123 sorts the scored coherent clusters 1121 from step 1119.Starting from the top of the list 1125 of sorted coherent clusters, step1127 then fits a circle to points that form the segments 1109 in eachcoherent cluster 1121. Step 1127 produces an initial boundary 1129 forthe pupil image 1003.

Step 1131 searches along the initial pupil boundary 1129 to generate alarger number of new more accurately positioned sparse edge points 1133corresponding to the pupil. The higher accuracy in step 1131 may beachieved by positioning rectangular search windows perpendicular to theinitial pupil boundary. In step 1135, a circle is fitted to the new setof sparse edge points 1135 to generate a refined boundary 1137 for thepupil image 1003. The pupil boundary 1137 provides a basis fordetermining a position for the center 1138 of the pupil image 1003.Accordingly, the algorithm determines data for the pupil image 1003.

As described previously, the pupil boundary 1137 also corresponds withthe inner boundary of the iris image 1002. With the inner boundary 1137of the iris image 1002 now determined, FIG. 11 further illustrates step1139 which searches for two edge points 1141 which estimate the outerboundary of the iris image 1002. This outer boundary is quasi-concentricwith the iris inner boundary 1137. Step 1139 searches for alight-to-dark edge to the left of the pupil image 1003 and for thedark-to-light edge to the right of the pupil image 1003 by using amodified normalized correlation. The iris outer edges are somewhatsymmetric with respect to the center 1138 of the pupil image 1003.Search windows 1140 employed by step 1139 may be long and narrow withlongitudinal axes which originate from the pupil center and are angleddownward from the horizontal by a predefined angle, e.g. 15 to 20degrees, to avoid interference with other features, such as eyelashes,near the top of the eye image 1001. FIG. 12 illustrates two angularrectangular search windows 1140 for edge detection using modifiednormalized correlation.

Using two edge points 1141 found in step 1139 and the pupil center 1138,step 1143 determines a circle corresponding to an initial outer boundary1145 of the iris image 1002 with a center which coincides with that ofthe pupil image 1003. Similar to steps 1131 and 1135 which refine theinitial pupil image boundary 1129, steps 1147 and 1151 refine theinitial outer iris boundary 1145. In particular, step 1147 generates alarger number of new sparse edge points 1149 by searching along theinitial outer iris boundary 1145, and step 1151 fits a circle to the newset of sparse edge points 1149 to obtain a refined outer iris boundary1153. Accordingly, an annular region 1154 in the eye image representingthe iris image is defined by the refined outer pupil boundary, or inneriris boundary, 1137 and the refined outer iris boundary 1153.

The accuracy of the iris segmentation technique just described may beenhanced by intermediate verification steps. For example, the techniquemay check for the presence of an iris image surrounding a candidatepupil cluster before accepting the candidate pupil cluster as the pupilimage.

As discussed previously, the digital processing algorithms 1010 may alsoinclude an eyelash/eyelid segmentation algorithm 1013. As an example,FIG. 11 further illustrates step 1155 which computes the magnitude ofocclusion 1156H of the iris image from a Cartesian image. In otherwords, step 1155 determines how much of the iris image 1002 isrepresented in the calculated annular region 1154 between the calculatedinner iris boundary 1137 and outer iris boundary 1153. FIG. 13illustrates the step 1155 in further detail. Step 1155A analyzes theimage intensity distribution in two relatively small areas in theannular region 1154 and slightly below the pupil. In general, theseareas are least affected by extraneous occluding features, such aseyelids, so the image intensity distribution in these small areastypically represent an image intensity corresponding to the iris.Therefore, iris intensity thresholds 1156A may be derived from the imageintensity distribution in these small areas. Step 1155B applies thederived intensity thresholds 1156A to detect “out-of-range” pixels 1156Bto find areas of occlusion, generally above and below the pupil. Step1155C then employs narrow search windows to the left and to the right ofthe calculated annular region 1154 to detect edge points 1156Crepresenting eyelids and/or eyelashes. Step 1155D filters out outliersand reduces the sets of edge points to edge points 1156D that correspondto the imaginary boundaries for the occluded regions in the eye image1001. In step 1155E, the boundary edge points 1156C, as well asintensity points, are divided into classes corresponding to the uppereyelid and the lower eyelid. Step 1155F then fits second orderpolynomials to the two classes, or groups, 1156D corresponding to theupper eyelid and the lower eyelid. Step 1155G then finds intersections1156G between the polynomial fits 1156F with inner iris boundary 1137and outer iris boundary 1153. Finally, the area 1156H of the occludedregions of the iris image is computed in step 1155H by integratingpixels between the corresponding curves defined by the polynomial fits1156E, inner iris boundary 1137, and outer iris boundary 1153.

For eyelid/eyelash segmentation, some embodiments may employ a two-stagetechnique that first applies coarse detection in a first stage and afine-scale mask generation in a second stage. Coarse detection providesa fast technique which can be employed real-time to measure roughly howmuch the upper and lower eyelids and eyelashes cover the iris in thecaptured eye image. In particular, coarse detection is able to providefast and efficient results by only testing the areas in the image thatare most susceptible to eyelid/eyelash occlusion. As such, the firststage may advantageously be employed at the time of image capture toreject quickly the images with low iris image quality resulting fromheavy eyelid/eyelash occlusion. Indeed, the software approach ofembodiments of the present invention attempts to capture the bestpossible images of the subject so biometric identification can beperformed with more precision.

On the other hand, the second stage is a slower but more accuratetechnique employing pixel-wise mask generation. In pixel-wise maskgeneration, every pixel in an unwrapped image is tested, or measured, todetermine whether the pixel is a part of the iris image or whether it isnoise associated with an eyelid or eyelash image. Accurate pixel-wisemask generation may be applied more appropriately at the time ofmatching. In some embodiments, the first stage may apply the sametechnique as the second stage, but in a faster, more selective manner.As such, FIG. 14 illustrates a further embodiment of a digitalprocessing algorithm 1013 which may be employed for two-stageeyelid/eyelash segmentation.

As shown in FIG. 14, the exemplary embodiment first computes a set oftraining histograms from an unwrapped image from regions empiricallybelieved to be free of occlusion. Test histograms are then be computedfrom neighborhoods of all pixels and tested for dissimilarity inrelation to the training set histograms. Thus, this embodiment employsiris intensity modeling from regions free of eyelid/eyelash occlusion asa basis for determining whether other regions of the eye image 1001correspond to eyelid/eyelash occlusion.

Referring to FIG. 14, data regarding iris segmentation, for exampleannular region 1154 as determined by SPM, are initially received in step1201. In step 1203, the annular iris region 1154 is unwrapped into arectangular image 1205 of fixed size, e.g. 512×64, using a polarunwrapping technique. FIG. 15A illustrates the annular iris region 1154calculated for captured eye image 1001, while FIG. 15B illustrates therectangular image 1205 which results from step 1203.

Step 1207 then determines training set histograms 1209 in the unwrappedimage for regions that are empirically and anatomically observed to befree of eyelid/eyelash occlusion. For example, in one particularembodiment, 8 training set histograms with 32 bins per histogram may becomputed from non-overlapping rectangular regions from the unwrappedimage around 90° (W/4) and 270° (3*W/4) locations. The rectangular areaof width 40 pixels and height 8 rows in the lower eyelid region oneither sides of the upper eyelid region is combined to generate each ofthe 8 training set histograms. The choice of parameters ensuressufficient image data points per histogram. The raw training sethistograms are then normalized to convert them into probabilitydistributions.

Once the training histograms 1209 have been computed in step 1207, a setof test points 1213 within the eye image 1001 may be selected in step1211 to form the basis for defining test regions 1217 in step 1215. Forinstance, step 1215 may define the test regions 1217 as widths of pixelscentered at the test points 1213 at each row of the unwrapped image1205. The algorithm determines whether these test regions 1217 are partof an eyelid/eyelash occlusion. In particular, in step 1219, a testhistogram 1221 is computed for each test region 1217. The normalizedtest histogram 1221 from each test region 1217 is then compared in step1223 against all the training set histograms 1209, one by one. Thecomparison is based on histogram similarity computed by histogramintersection score 1225 defined as:

${S_{j,k} = {\sum\limits_{b = 1}^{B}{\min\left( {{Ts}_{b}^{j},{Tr}_{b}^{k}} \right)}}},$where B represents the total number of bins in the binned histogramrepresentation (e.g., 32 bins in an exemplary embodiment), Ts_(—) ^(j)corresponds to the test histogram at j^(th) test set level, Tr_(—) ^(k)corresponds to the k^(th) training set histogram. For example, if thetraining set 1209 contains 8 histograms of size 8×40 each, variable kgoes up to 8; on the other hand, if the test histograms are of size1×40, the variable j goes up to 64. An overlap threshold 1226, e.g. 33%corresponding to a normalized score of 0.33, between normalizedhistograms may be defined. Step 1227 determines if the similaritybetween a particular test histogram 1221 and all the training sethistograms 1209 is less than the threshold 1226. If so, the test isdeclared to fail, indicating that the test region 1217 belongs to aneyelid/eyelash occlusion.

It is duly noted that eyelid/eyelash segmentation in this embodimentemploys gray-level histograms merely as a tool for density estimation ofthe iris/non-iris intensities in an unwrapped iris image. The techniquedescribed above is does not depend on the use of histograms. Indeed,more sophisticated density estimation approaches, especially KernelDensity Estimation (KDE) may also be used. KDE is discussed in E.Parzon, “On estimation of a probability density function and mode”,Annals of Mathematical Statistics, 33:1065-1076, 1962, which is entirelyincorporated herein by reference. To compare histograms and KDErepresentation in an example, one may consider the generation of atraining set histogram. Let the gray-scale pixel values corresponding tothe region in unwrapped image under analysis be {x₁, . . . , x_(n)}, nbeing the total number of pixels in the region. For a 32-bit histogram,the range of a gray-level pixel value for 8-bit image is 0-255inclusive. This range is divided into 32 cells of equal dimension, soevery 8th gray-scale value quantizes to a different bin. A histogramestimates the density of underlying gray-scale values as:

${\hat{p}(x)} = \frac{n_{j}}{\sum\limits_{j = 1}^{N}n_{j}}$where n_(j) represents the number of pixels in the region beingcurrently analyzed whose gray-scale intensities lie in the range ofj^(th) histogram cell, and N represents the total number of histogramcells, 32, in our case. Also, note that

${\sum\limits_{j = 1}^{N}n_{j}} = {n.}$Using KDE framework, the density estimation from the pixels of sameimage region is computed using a kernel of bandwidth (spread orsmoothing parameter, h) as:

${\hat{p}(x)} = {\frac{1}{nh}{\sum\limits_{i = 1}^{n}{\frac{1}{\sqrt{2\pi}}{\exp\left( {{- \frac{1}{2}}\left( \frac{x - x_{i}}{h} \right)^{2}} \right)}}}}$In the above representation, Gaussian kernel of width h is used toderive a smoothed representation of the underlying density. Using thisapproach, a continuous and smoothed density estimate is derived which,in some cases, might result in a better representation of thetraining/test regions as compared to representation using histograms.

In some instances, the technique based only on gray-scale imageintensity might fail to give sufficient results, particularly for theeyelid region, because dark eyelashes combined with bright eyelid maycreate a histogram similar to that of iris texture pattern. Thus, asfurther illustrated by FIG. 14, another level of testing in step 1231may be employed by checking the edge content in the particular testregion 1217. For instance, if step 1227 determines that the comparisonscore 1225 is not less than the overlap threshold 1226, step 1231determines the number of edge pixels 1233 in the test region 1217. Edgedetection may be performed using Canny edge detector with lowerthreshold of 0, upper threshold of −1 and using a Gaussian window ofsigma 1. If the number of edge pixels 1233 in the test region 1217exceeds a threshold 1234, e.g. 10%, of the total pixels in the testregion 1217, the test region 1217 is marked as an eyelid occlusion.

As discussed previously, this eyelid/eyelash segmentation technique maybe employed for first stage coarse detection, which only tests the areasin the image that are most susceptible to eyelid/eyelash occlusion. Inan example embodiment, once the training set histograms 1209 have beendetermined in step 1207, the set of test points selected by step 1211may include two vertical columnar areas 1213 which lie at 180° (W/2) and360° (W) locations corresponding to upper and lower eyelidsrespectively. Moreover, step 1215 may define the test regions 1217 aswidths of 40 pixels centered at the test points 1213 along the twovertical columnar areas, at each row of the unwrapped image 1205. Thealgorithm determines whether these test regions 1217 are part of aneyelid/eyelash occlusion by computing a test histogram 1221 for eachtest region 1217 as described previously. FIG. 15C illustrates exemplaryresults of coarse, or fast, detection to determine a coverage measurecorresponding to eyelid/eyelash occlusion.

In addition, the histogram test data and the edge detection dataproduced by the algorithm of FIG. 14 complement each other and may beused to determine a coverage measure for first stage coarse detection.In other words, this embodiment provides the total number of points thatpass the test of belonging to the iris image 1002. The coverage measureat this fast detection stage, C_(D), is then computed as:

${C_{D} = \frac{N_{i}}{N_{T}}},$where N_(i) is the number of pixels belonging to unoccluded iris patternand N_(T) denotes the total number of points tested. ANSI specificationsfor iris image capture and interchange dictate that the minimum coveragemeasure (the ratio of noise pixels to total iris pixels) is to be 0.7 ona scale of 0 (no iris texture pixel) to 1 (no noisy pixels from eyelidor eyelash). As such, the first stage coarse detection may reject theframes with a coverage measure falling below 0.7. In this way, allsubsequent stages of iris recognition work on frames pass the ANSIspecification of good iris images in terms of eyelid and eyelash noise.

As described previously, the eyelid/eyelash segmentation technique ofFIG. 14 may also be employed for second stage pixel-wise maskgeneration. In an example embodiment, once the training set histograms1209 have been determined in step 1207, the set of test points for step1215 includes every pixel in the unwrapped image. Step 1215 may definethe test regions 1217 as areas of 8×80 centered at each pixel. Thealgorithm determines whether these test regions 1217 are part of aneyelid/eyelash occlusion by computing a test histogram 1221 for eachtest region 1217. If a test histogram 1221 for a test point 1217 has anintersection similarity, i.e. comparison score 1225, with any one of thetraining set histograms 1209 that is above the threshold 1226, the pixelat the test point is marked as an iris texture pixel. To adjust for theiris-like gray-scale histogram presented at the region of the eyelid,the edge data from Canny edge detector is used as in the first stage.This gives a binary mask with pixels corresponding to occlusion marked“1” and iris texture pixels marked “0”. The coverage measure from themask, C_(M), is then computed as:

${C_{M} = \frac{N_{i}}{W \times H}},$where N_(i) is the number of pixels belonging to un-occluded irispattern, and W and H denote image width and height. Finally this binarymask is used in iris code generation for only the occlusion-free pixels.This way, the effect of noise due to eyelid and eyelash occlusion isavoided in the resulting iris code. FIG. 15D illustrates exemplaryresults of pixel-wise mask generation.

Accordingly, the embodiment presented in FIG. 14 employs the gray-scaleintensities of iris and non-iris regions for eyelid/eyelashsegmentation. To demonstrate the effectiveness of the algorithm of FIG.14, FIG. 16 illustrates an example of the absolute difference inpercentage coverage measure between ground truth and fast, or coarse,detection (marked as “Fast Detect”) and between ground truth andpixel-wise mask generation (marked as “Mask”).

An alternative embodiment for providing an accurate mask generationprocess, illustrated in FIGS. 17 and 18, models the texture pattern ofthe iris regions more explicitly to distinguish it from non-iris regionsin the unwrapped image. In particular, the texture modeling in thisalternative embodiment is performed using a bank of log-Gabor filters togenerate a texture representation based on the phase congruencyfeature-space. This feature space analyzes the spectrum of the givenimage at various frequencies to compute the alignment of phase at afeature location. For example, if there is a step edge present in animage, then in the frequency spectrum of the image, different phasecomponents have a zero-crossing at the location of the edge point. Thisobservation motivates using congruence of a multitude of phasecomponents to represent dominant features in the image. The computationof phase congruency using log-Gabor wavelet filter banks is discussed inPeter Kovesi, “Invariant Measures of Feature Detection”, Ph.D. thesis,The University of Western Australia, 1996, which is incorporatedentirely herein by reference.

Referring to FIG. 17, data regarding iris segmentation, for exampleannular region 1154 as determined by SPM, is initially received in step2002. In step 2003, the annular iris region 1154 is unwrapped into arectangular image 2004 using a polar unwrapping technique. Using thepolar unwrapped image 2004, a phase-congruency map image 2006 isgenerated using a bank of log-Gabor filters in step 2005. This bankconsists of a set of orientation and scale filters tuned to variousfrequencies in order to generate sharper response at particular imagefeature points. Indeed, the image features that help distinguish an irispattern from non-iris patterns can occur at various orientations and atvarious sizes. This method captures these features at variousorientations and various sizes and combines the results from all thefilters to generate iris texture pattern representation for use in imagemask generation. The resulting phase congruency map image values rangefrom 0 to 1 (as opposed to 0-255 for 8-bits gray-scale images).

As shown further in FIG. 17, the embodiment creates a weightedtexture-intensity image 2010 for dark areas and a weightedtexture-intensity image 2013 for light areas 2013 via process 2007 andprocess 2014, respectively. The phase congruency image 2006 and irisintensity image in the polar unwrapped image 2004 are used to generatethe images 2010 and 2013. Weighted texture-intensity image for darkareas, 2010, is generated in step 2008 as:

${I^{d}\left( {x,y} \right)} = {{P\left( {x,y} \right)} + {\varpi^{d}\left( {1 - \frac{I\left( {x,y} \right)}{255}} \right)}}$where I_(d)(x,y) represents a pixel at x,y location in thetexture-intensity weighted image 2010 for dark areas, P(x,y) representspixel at the same location in the phase congruency image 2006, ω ^(d)represents dark areas weight factor 2009 (an exemplary value is 0.625),and I(x,y) represents a pixel at same location in iris intensity image.The weighted texture-intensity image for light areas, 2013 is generatedin step 2011 as:

${I^{I}\left( {x,y} \right)} = {{P\left( {x,y} \right)} + {\varpi^{i}\left( \frac{I\left( {x,y} \right)}{255} \right)}}$where ω ¹ represents a weight factor 2012 for light areas (an exemplaryvalue is 0.5).

From the weighted texture-intensity images 2010 and 2013, the process2015 generates an initial mask image 2024. Similar to the embodiment ofFIG. 14, step 2017 generates a training set vector from designatedareas, deemed free of eyelid/eyelash segmentation. In addition,processing in step 2017 may be limited to weighted texture-intensityvalues that correspond to pixels having original image intensitieswithin a valid intensity range 2016. In step 2018, the training vectoris sorted on texture-intensity values. Dynamic upper and lowerthresholds 2021 for hysteresis filtering are the generated from thesorted vector 2019 and fixed upper and lower thresholds 2020 (e.g., 97%and 93%). Step 2022 employs the resulting upper and lower thresholdscomputed in step 2021 to apply a hysteresis filter. The hysteresisfiltering results from dark and light areas are then combined in step2023 to generate a combined initial mask 2024.

In particular, the process 2015 generates an unwrapped binary image mask2024, in which every pixel is marked “1” for occlusion and “0” for irispixel. The binary image 2024, however, may contain undesirable levels ofeyelid/eyelash regions that are marked erroneously as iris regions, orvice versa. In one case, the binary mask image 2024 may contain holescorresponding to imperfect occlusion marking. This phenomenon may resultfrom the fact that the iris texture pattern is combined with gray-scaleintensity at every pixel to generate a fused representation of thepixel. Thus, at some eyelid/eyelash pixel locations, the local structuremight resemble that of the combined iris texture and intensityrepresentation. In the opposite case, the unwrapped mask image 2024 maycontain small regions inside the iris region that are improperly markedas eyelid/eyelash occlusions. This phenomenon occurs when a largepopulation of human irises is involved and certain local structuresarise which do not represent a pattern typical of most of the irises.

As illustrated further in FIG. 17, the embodiment presented herein maycorrect for such imperfections in the mask image 2024 by employingprocess 2025 which involves connected components labeling.Alternatively, the process 2025 may employ the histogram techniquedescribed previously with reference to the embodiment of FIG. 14.Referring to FIG. 17, the steps 2027 and 2031 isolate the upper eyelidregion 2028 and lower eyelid region 2032, respectively. The half-imageshaving a size of (W/2)×H are processed separately in steps 2029 and 2033using the process 2037 illustrated in FIG. 18. If, for example, theannular iris image 1154 is unwrapped into 512×64 pixels, the rectangularnon-overlapping regions representing the two independent halves of size256×64 are processed separately. The upper eyelid mask 2030 and thelower eyelid mask 2034 are then combined together in step 2035 togenerate a final mask image 2036.

The half-image processing shown in FIG. 18 first receives a half region,i.e. the upper eyelid region 2028 and lower eyelid region 2032, in step2038. Step 2039 then performs connected components labeling onhalf-image binary mask received in step 2038. In particular, step 2039generates a set of regions (groups of pixels) that are marked as “1”,i.e. occlusions, and that share spatial boundaries based on 8-connectedneighborhood. As such, step 2039 produces a connected components labeledimage 2040. Step 2041 then computes the probability that each connectedregion is an eyelid component. This computation is based on the factthat the eyelid region presents the pattern of a large area componentwith a high value for its maximum y-coordinate. As such, the eyelidcomponent may be inferred as the solution to the following

$C_{lid} = {\underset{i}{argmax}\left( {{\alpha\;\frac{Y_{{ma}\; x}^{i}}{H}} + {\beta\;\frac{A^{i}}{\left( \frac{W \cdot H}{2} \right)}}} \right)}$where α and β denote weights for y-coordinate and area components,Y_(max) ^(i) and A^(i) represent the maximum y-coordinate and area ofthe i^(th) connected component, and W and H represents the width andheight of unwrapped image respectively. Thus, step 2041 generates thebest eyelid candidate 2042. As a result of the iris unwrapping, theeyelid component is constrained to have a maximum y-component close tothe unwrapped image height within a tolerance Δ_(y) of the unwrappedimage height. If this final condition does not hold, the best eyelidcandidate 2042 from step 2041 is invalidated in step 2043. Inparticular, step 2043 generates a flag 2044 that indicates the presenceor absence of the eyelid component.

As FIG. 18 also shows, the process 2037 cleans up regions inside theiris pattern which are marked as occlusions. For example, the connectedcomponents that are either too small or too far from the eyelidconnected component are marked as iris regions. Initially if step 2048determines that the eyelid is not present according to the flag 2044,all pixels corresponding to the connected components are rejected fromthe occlusion mask and marked as iris pixels. Indeed, if the largestform of occlusion, an eyelid, is not present then the chance for thepresence of other occlusion significantly reduces. If an eyelid ispresent according to flag 2044, the occlusion mask pixels are notchanged at step 2048. In this case, step 2046 then computes theHausdorff distance between each connected component set from the eyelidconnected component set. The resulting Hausdorff distance 2049 for thecurrent connected component is then employed to determine if the currentcomponent should to be rejected from the occlusion mask. The currentconnected component is rejected if the connected component is more thanΔ_(H) pixels away from eyelid component according to step 2051.Otherwise, if the connected component is not more than Δ_(H) pixels awayfrom eyelid component, the occlusion pixels are not changed at step2051. The current connected component is also rejected from theocclusion mask if the area 2050 of the connected component determined instep 2047 is less than a threshold according to step 2052. Otherwise, ifthe area 2050 is not less than the threshold Δ_(A), the occlusion pixelsare not changed at step 2052. Accordingly, pixels in the iris regionmarked as occlusions are rejected from the occlusion mask and correctlymarked as iris pixels.

As further shown in FIG. 18, once the cleaned up, or updated mask imageis produced, mask post-processing is launched for the current half ofthe unwrapped iris image. In this phase, step 2056 fills in the holes inthe half unwrapped iris image mask. Step 2056 may be required becauseexcessive eyelid occlusion may occur and render false the assumptionthat eyelashes and the dominant portion of eyelid present a texture andintensity level different from the iris region. Therefore, step 2056 mayiterate over all columns of the updated binary mask and identify thecolumnar strips which are bound on the lower and upper ends by pixelsmarked as occlusions, i.e., “1,” but have some pixels marked as iris,i.e., “0”. After identification of these upper and lower bounds and gapsin the columnar strips, step 2056 marks the pixels between boundingpoints as “1” to indicate an occlusion pixel. Step 2056 produces ahole-filled mask image 2057. In addition, step 2059 identifies isolatedpixels that have intensity above a maximum intensity threshold andqualify as reflection noise. Such pixels are marked as occlusion noisein step 2059, which produces a reflection-filled mask image 2060.Finally, step 2061 identifies pixels that have intensity below a minimumintensity threshold and qualify as pupil pixels. Such pixels are markedat occlusions by step 2061, which produces a pupil-filled mask image2062. It is noted that the presence of pupil pixels may be drawn intothe iris region during the segmentation and unwrapping process. Theresulting mask images 2057, 2060 and 2062 are combined in step 2058,which generates the final mask image 2063 for the half unwrapped irisimage received in step 2038.

Example results of the mask generation and outlier rejection process areillustrated in FIGS. 19A-H. FIGS. 19A, C, E, and G illustrate fourunwrapped iris images 2101, 2103, 2105, 2107, while FIGS. 13, D, F, andH respectively illustrate their corresponding masks 2102, 2104, 2106,2108.

As discussed previously, with reference to FIG. 10, the digitalprocessing algorithms 1010 may also include an iris focus measurementalgorithm 1014. Advantageously, the effect of eyelashes, the texture ofthe iris, and/or other noise are minimized with both embodiments of irisfocus measurement algorithms, shown in FIGS. 20A and 20B.

FIG. 20A illustrates one embodiment of an iris focus measurementalgorithm 1014, which employs a gradient technique across the iris/pupilboundary. Using the pupil boundary 1137 and the pupil center 1138, forexample as determined by SPM, step 1301 determines the gradientmagnitude 1303 across the iris/pupil boundary 1137 in a radial directionwith respect to the pupil center 1138. Step 1305 then creates ahistogram of the gradient magnitude 1303. Using the histogram 1307, the90 percentile value of the gradient magnitude 1303 is calculated in step1309 as the focus measure 1311, which may be normalized to a scale of 0to 1, 0 to 100, etc. In particular, the gradient method of FIG. 20Aminimizes the effect of noise by using the 90 percentile value of theaccumulated histogram. The use of a 90 percentile value of the magnitudehistogram has been provides a reliable focus measure.

FIG. 20B illustrates another embodiment of an iris focus measurementalgorithm 1014, which employs the lighting reflection from imagecapture. As described herein, to capture an eye image, embodiments ofthe present invention employ a camera sensor and a light setting whichproduce a light reflection 1313. The measure of focus is in proportionto the size and the clarity of the light reflection 1313. Thus, step1315 measures the size 1317 of the light reflection 1313. As the size1318 of light reflection at the best focus point is ascertainable, step1319 determines a normalized value for the focus measure 1321 based onthe ratio of the measured size 1317 and the best focus size 1318. Inparticular, this embodiment avoids the effect of noise by using theactual lighting reflection. The application of this embodiment may varyaccording to the acquisition, because the reflection size varies fromdevice to device.

With respect to other approaches for obtaining a focus measure, it hasbeen observed that during actual acquisition of iris images, the use ofimage frequency based focus measure (for example, as described in U.S.Pat. No. 6,753,919) disadvantageously obtains the best focus images foreyelashes or eyebrows and not the iris, because eyelashes and eyebrowsmay contain high frequency content. In addition, it has also beendiscovered that the use of the total magnitude of gradient (for example,as described in U.S. Pat. No. 5,404,163) instead of radial magnitude issensitive to the pattern of the iris and thus not usable for iris focus.Furthermore, Int'l Pat. Pub. WO 99/27845 describes the use of a radialgradient where the division of the average of the magnitude divided thestep size provides the focus measure, but this technique has been foundto be sensitive to noise.

In one aspect, the digital processing algorithms 1010 enable the threedimensional position of the iris for the left eye and/or the right eyeto be determined with respect to the multimodal ocular biometric device.For example, information regarding position along X- and Y-axes may bedetermined from the pupil segmentation algorithm 1011 while informationregarding position along the Z-axis may be determined from the irisfocus measurement 1014. Accordingly, as described further below, suchdata may be used to determine whether the iris images captured by thesensor 110 are of acceptable quality and should be used for furtherbiometric evaluation.

For some embodiments of the present invention, aspects of capturing aniris image are described with reference to FIGS. 21 and 22. As describedpreviously with reference to FIG. 1, a plurality of image frames may bereceived from camera sensors 110 which capture light from an eye,particularly the iris, which reflects the emitted light from theillumination source 120. Accordingly, an auto-capture process 1400, asshown in FIG. 21, may extract the required biometric information fromthe sequence of frames. Step 1410 sequentially receives each of theimage frames 1405 into memory in the form of an image bitmap. In step1420, a ready signal is triggered for processing of the image frame whenthe bitmap transfer from the camera, e.g. sensor 110, is complete.

In an alternative embodiment, numbered image frames 1405 are transferredfrom the camera to a circular memory buffer in the form of imagebitmaps. This circular buffer is continually updated as image frames1405 are transferred from the camera. Initially, processing is startedwhen the first image frame 1405 is read into the circular memory buffer.Processing threads then transfer the latest image bitmap into memorylocal to the thread. The processing thread then processes the imagebitmap as described below. On completion of analysis, if processing hasnot been terminated, the thread then transfers the next image bitmapfrom the circular buffer to memory and repeats processing steps for thisimage. In this manner, it is possible for the invention to miss or dropvideo frames from processing. In other words, as the processing stepsare applied to a single image bitmap, the circular camera buffer may beupdated a number of times before the processing thread transfers thelatest image bitmap from the circular buffer. However, it is the goal ofthe present invention to drop as few video frames as possible. A furtheralternative embodiment includes an acquisition thread that controls thetransfer of image bitmaps from the circular image buffers of each camerato the processing thread. For systems with multiple cameras, such as themultimodal biometric system 10 and other embodiments described herein,multiple circular image buffers for a system may be employed where eachcircular image buffer is controlled by acquisition threads that feedimages to single or multiple processing threads. The processing threadsmay represent different processing steps designed for differentpurposes.

Referring again to FIG. 21, as a part of a segmentation test, step 1430tries to identify the inner and outer boundaries of the iris image, forexample, with the pupil and iris segmentation algorithms 1011 and 1012described previously. In addition, step 1430 tries to identify theboundary of the upper eyelid, for example, with the coarseeyelid/eyelash detection algorithm 1013 described previously. If boththe inner and outer iris boundaries are identifiable, the image framepasses the segmentation test and further processing occurs with step1440. Otherwise, the process loops back to step 1420 where the processstarts again with the next image frame. In an alternative embodiment, nosegmentation test is executed, so step 1420 proceeds on to step 1440directly. In another alternative embodiment, the test of step 1430 onlytries to identify the inner iris boundary, and not the outer irisboundary or the upper eyelid boundary. In yet another alternativeembodiment, step 1430 identifies both the inner and outer irisboundaries, but not the upper eyelid boundary.

Step 1440 executes an assessment of the image quality with an imagequality test. The details of the image quality test are furtherillustrated in FIG. 18. Accordingly, the image frame is received in step1440A, and the step 1440B determines the pupil/iris boundary 1440C, forexample, with the pupil and iris segmentation algorithms 1011 describedpreviously. Step 1440D calculates an intensity contrast in an areadefined by the pupil/iris boundary according to the gradient techniqueof the iris focus measure algorithm 1014 shown in FIG. 20A. As describedpreviously, the result of applying the gradient technique is called afocus measure, referenced in FIG. 22 as 1440E. In step 1440F, the focusmeasure 1440E is compared to a predefined threshold 1440G. If thecalculated value exceeded the threshold the image passes the imagequality test as shown in step 1440H; otherwise, it fails as shown instep 1440I.

Other embodiments may employ alternative image quality tests. An exampleuses contrast and/or texture within the identified area of the iris. Forinstance, a high-pass filter could be used to quantify high frequencycomponents in the iris with the idea that a good quality iris imagecontains more high frequency components than a lower quality iris image.

Referring again to FIG. 21, if the image frame passes the image qualitytest in step 1440, the data corresponding to the image frame 1405 isadded to an image data cache 1455 in step 1450. This data includes theimage frame in the form of a bitmap, its image quality score, andassociated information and any segmentation results calculated in step1430. This data may also be referred to as an image'sacquisition-result. When this record is added to the cache 1455, it isplaced in ranked order along with any records already within the cache.In other words, the cache 1455 holds a ranked queue of irisacquisition-results derived from the plurality of images processed thusfar. The iris acquisition-results may be ranked according to criteria,such as the focus score.

A maximum of M (M≧1) iris acquisition-results are held in the cache.This number may change depending on whether a user is being enrolled,verified, or identified according to the captured biometric data. If, instep 1460, the cache 1455 already contains the maximum permitted numberof iris acquisition-results, the current iris acquisition-resultreplaces the lowest ranking iris acquisition-result in the cache 1455 ifthe current iris acquisition-result ranks higher. The process then loopsback to step 1420 where the analysis of a new image frame 1405 starts.

However, if the image frame 1405 fails the image quality test in step1440, the process moves on to step 1460. Here the number of irisacquisition-results in the cache 1455 is checked against a definedthreshold, N (M≧N). If the cache 1455 does not contain enough irisacquisition-results, then necessarily not enough image frames have thusfar passed both the segmentation test, in step 1430, and the imagequality test, in step 1440, and the processes loops back to step 1420where the analysis of a new image frame 1405 starts. If, however, thecache 1455 contains enough records then the process moves onto step1470.

At step 1470 the top O (N≧O) ranked iris acquisition-results are removedfrom the cache and, in step 1490, a “successfully acquired” signal issent to controlling software to indicate that acquisition has succeeded.The auto-capture process 1400 is halted and the process continues to thefinal encoding step 1500.

At any point during the auto-capture process 1400, a timeout signal, instep 1480, can be received from the controlling software and theauto-capture process 1400 is halted. The processing thread is permittedto continue through to step 1440, if necessary. If the image frame 1405passes the image quality test of step 1440, the process moves onto step1450 and then is transferred to the cache 1455. If the image frame failsstep 1440 the process moves directly to step 1470.

If fewer than O results are contained in the cache 1455 after all imageframes have been analyzed or the auto-capture process 1400 has beenhalted by timeout in step 1480, the auto-capture 1400 has failed toextract the required information and a “failed to acquire” signal isreturned.

At step 1500, the extracted iris acquisition-result(s) are encoded intoa biometric format. If the encoded results are being used for biometricverification, the results proceed to matching modules. If the encodedresults are being used for biometric enrollment, the results can becompressed and/or encrypted for future use.

An alternative embodiment may analyze time contextual information duringthe image quality test in step 1440. For example, if an image framepasses the image quality test in step 1440, it then undergoes atime-contextual test. In other words, if the segmentation and/or imagequality test results show a significant disparity between a currentimage frame and the last image frame, the current image fails thetime-contextual test and is not considered or added to the irisacquisition-result cache in step 1450.

Accordingly, the digital processing algorithms 1010 illustrated in FIG.10 may be employed to evaluate whether a captured image should beretained and to identify segments of the captured image from which datacan be extracted for biometric enrollment or verification.

With reference again to FIG. 1, once iris image information has beenobtained and processed as described above, the retina illumination mayemploy a tracking system to illuminate the optic nerve head of theretina. For instance, arrays of LED's 220 at a wavelength of 880 nmspaced 1 mm apart are aligned to 1 mm diameter and 10 mm long hollowtubes. The hollow tubes create a homogenizing waveguide for the lightemanating from them. Only a single element of the array is illuminatedat a time corresponding to the determination of the pupil's position inspace, as determined by the digital processing algorithms 1010 describedpreviously. As such, analysis of the iris image yields pupillarypositional information that may be employed to determine illumination ofthe corresponding retina. In other words, the pupil's position is usedto determine which LED 220 in the array aligns most optimally with theretina and should be activated for illumination of the retina. Referencenumeral 225 in FIG. 1 illustrates a diffuser, which is placed over theends of the tubes to create a 1 mm spot from the active LED 220.

Alternatively, reference numeral 225 may refer to an LCD shutter, whichcan create a similar 2-dimensional series of singly activatedilluminators that are 1 mm in diameter and imaged to the eye. Dependingon the determination of the pupil's position in space, the LCD shutter225 allows light from the illumination source 220 to pass through anappropriate section of the LCD device 225 to illuminate the retina. Asfurther alternatives, scanning micro-optics or holographic elements mayalso be employed.

The light from the LCD shutter/diffuser/micro-optics 225 reflects off apolarizing beamsplitter (PBS) 230 creating S polarized light. This lightis then imaged by the aspheric objective lens 240, through a long passplastic sheet filter with a 780 nm cutoff wavelength, to a 2 mm spotjust before the nominal position of the cornea. The angle of the lightentering the pupil is nominally 15.5 degrees temporal to and 1.5 degreesinferior to the line of sight of the user. The spot diameter is chosento be smaller than the pupil so that light does not scatter off itsedges causing excess noise in the retina image. The divergence of thelight is approximately 10 degrees half angle. This allows for imaging ofa large enough FOV to obtain a suitable retina image for patternrecognition. The retina image consists of the blood vessel patternemanating from the optic nerve head. Absorption of the light byhemoglobin and oxyhemoglobin in the blood creates the outline of theblood vessel pattern. Demarcation of the optic nerve head may or may notbe discernable. The LED's have three pulse duration settings that arecycled through (exposure bracketing) so as to accommodate forreflectance differences of the retina in the general population.

Light reflecting off the retina passes back through the long pass cutofffilter. This filter prevents ambient visible light from entering theimaging system and creating noise in the image. It also hides theimaging optics from the user. The light is then collected by theaspheric objective lens 240 to produce a real image just before thepolarizing beamsplitter 230. This real image is then imaged though thePBS 230 allowing only P polarized light to pass. The purpose of the PBS230 is to increase the signal to noise ratio of the signal by rejectingany S polarized light reflected back through the system from otheroptical surfaces. An imaging lens followed by a cubic phase mask opticthen images the light onto a camera sensor 210. The camera sensor 210may be a CMOS detector with high sensitivity to NIR illumination. TheCMOS detector has square pixels, has a wide angle format, and has aglobal shutter.

The images of the retina are multiplied by specific digital filters.These filters are created for differences in dioptric power correction.The images are evaluated using a retina focus measure algorithm and theone with the highest contrast image is preferably utilized for biometricidentification. An example of a retinal focus measure algorithm isdescribed in application Ser. No. 11/785,924, filed Apr. 20, 2007, whichis entirely incorporated herein by reference.

The illumination for the iris may have a different wavelength from theillumination for the retina. In one embodiment of the present invention,the retina is illuminated with light of a first wavelength, the light ofthe first wavelength being reflected from the retina to the retina imagecapturing device. The iris is illuminated with light of a secondwavelength that is different from the first wavelength, the light of thesecond wavelength being reflected from the iris to the iris imagecapturing device. The first wavelength of light is selected to provideenhanced contrast between biometric features of the retina, such as aretinal vessel pattern, and the background in the captured image.Similarly, the second wavelength of light is selected to provideenhanced contrast for the biometric features of the iris.

If the iris illumination and the retina illumination occur at the sametime or in near time, however, the iris illumination can introduce noisein the retina signal, or vice versa. To avoid introduction of noisebetween the illumination of the iris and retina, dichroic optics can beemployed to allow wavelength separation from the different illuminationsources, where light of one wavelength is directed to one sensor whilelight of a second wavelength is directed to another sensor. Theillumination with special dichroic optics can be pulsed or run as acontinuous wave.

More advantageously, to eliminate the introduction of noise between theillumination of the iris and retina, the iris illumination and theretina illumination can be separated by pulsing the individual LEDs witha synchronized offset. For instance, the iris and retina cameras can runat 30 frames per second offset by half a frame (16.5 ms) with a shutter(global, rolling or global-rolling hybrid) of 10 ms. The pulses from theLEDs occur at 10 ms so that neither camera sees light from the otherillumination LEDs. The advantage of pulsing illumination with asynchronous offset is that it freezes motion, maximizes frame ratewithout having to use dichroics, and allows higher pulse energies whichreduces gain on the camera, thereby increasing image quality.Furthermore, pulsing illumination with a synchronous offset permits theuse of the same wavelength for the illumination of the iris and retina.

In general, both iris and retina illumination may use auto gain in orderto correct for the proper exposure for correction of reflectancedifferences of the iris and retina. Alternatively, both iris and retinaillumination bracketing (or exposure bracketing) may be used instead ofauto gain. In this alternative approach, two or more illumination powersettings are cycled through to bracket through all possible reflectancedifferences seen in the general population; for example: power setting 1(pulse 1)=10 units, power setting 2 (pulse 2)=12 units, power setting 3(pulse 3)=14 units, where cycle=pulse 1, pulse 2, pulse 3, pulse 1,pulse 2, pulse 3, . . . an so on. One could also do this by keeping thepower constant and cycling three different pulse durations; for example:pulse duration 1 (pulse 1)=10 units, pulse duration 2 (pulse 2)=12units, pulse duration 3 (pulse 3)=14 units, where cycle=pulse 1, pulse2, pulse 3, pulse 1, pulse 2, pulse 3, . . . an so on.

Accordingly, in the embodiment shown in FIG. 1, the iris illuminationcan advantageously be pulsed at less than half the frame rate of theiris and retina cameras. The frame rates for both cameras are identical.The image of the iris is analyzed with the pupil tracking and iris focusmeasure digital processing algorithms. The X₁, Y₁, and Z₁ positions ofthe pupil of the iris are calculated. The user must move through thenominal Z_(N) position of the system which establishes the absoluteposition of the user. Until that time, the system assumes a relativeposition of the pupil based on pupil size. Iris images that areadequately in focus are collected and analyzed appropriately. Asdescribed above, the LED's have three power settings that are cycledthrough (exposure bracketing) so as to accommodate for reflectancedifferences of the iris in the general population.

As described previously, the positional information of the pupil isutilized to select the addressable retinal illumination LED that willcleanly enter the pupil. The retina illumination LED is pulsed at half aframe out of phase from the iris illumination. The pulse duration isless than half the frame rate. As described above, by synchronizing theiris and retinal frame rates of the camera at half a frame rate out ofphase with each other and using short pulses, the full frame rate ofeach camera can be utilized while minimizing noise that may occurbetween the illumination of the iris and the retina. Illumination pulseswith shorter time frames freeze motion and increase image quality.

The present invention may also employ a retina auto focus mechanism,which corrects for changes in retinal focus due to differences inuncorrected dioptric power and allows any corrective optical devices tobe removed by the user. Corrective optical devices can cause aberrationsand glare. Several techniques may be applied to achieve retina autofocus.

As shown in the retina imaging system 200 of FIG. 2A, one technique forretina auto focus employs a motor 292 that moves the focus of the retinaimaging lens 290 in specific dioptric value increments, along the arrowA as shown in FIG. 2A. The system utilizes a retina focus measurealgorithm comparing successive positions. If the system remains out offocus, the system uses this comparison to determine the direction inwhich it should move.

As shown in FIG. 2B, another technique for retina auto focus employswavefront coding technology using cubic phase plate and signal analysis.FIG. 2B illustrates a retina imaging system 200B with an imaging lenswith a cubic phase plate, indicated by reference numeral 294. Contraryto the use of the motorized lens, there are no moving parts withwavefront coding. A cubic phase mask is placed in the system and thesystem is fully characterized with regard to dioptric power correction.Differences in dioptric power correction are calculated and specificdigital filters are created for each dioptric power. When an image istaken, each of the filters is convolved with the image and the one withthe highest contrast image is utilized. This configuration provides arobust system, which can be used at extreme temperatures, because thereare no moving parts.

As depicted in the retina imaging system 200C of FIG. 2C, a thirdtechnique for retina auto focus uses an electroactive optical element296, which is a liquid crystal sandwiched between two pieces of glasswith a specific electrode configuration on them. By activating theelectrodes with different either a positive or negative dioptriccorrection may be created. This can be a single device or a stack ofdevices to create larger dioptric correction.

While the auto focus systems above have been described in terms ofretina imaging, it is understood that such auto focus techniques arealso applicable to an iris auto focus system.

In general operation, the multimodal ocular biometric system accordingto the present invention may be handheld, but may also be attached to anarticulating arm, attached to or embedded into an immovable object suchas a wall, or adapted to an existing optical system such as a riflescope or tank periscope. As described further below, the system maypossess a simple fixation system, or interface, to position the user.For instance, with an exemplary handheld embodiment, the user picks upthe device and removes any eyeglasses the user may be wearing. The userthen identifies a fixation illumination source within the device andcarefully positions the device with respect to his or her face accordingto the fixation illumination source. As also described in anotherembodiment below, the outer housing of the device may be designed tohelp center the user as well as to provide light baffling of externalambient light.

With reference to FIG. 1, the user operates the image capture device 12by identifying the fixation light source 310 through the broadbandantireflection coated windows 330. The light from the source 310reflects off the beamsplitter and cold mirror 320. In a fixation system60 illustrated in FIG. 7, a circular target 62 with cross hairs 64 isviewed through an imaging lens with two illuminated bars 66 above andbelow the lens. The illuminated bars 66 are positioned at the exit pupilof the device 12. The bars 66 may include a diffusing light guide withcolored LEDs illuminating them. The circular target 62 is a reticulewith a diffuser and colored LEDs behind it. The user operates thefixation system 60 by moving the device 12 relative to his or her eyesto center the circle 62 between the two bars 64. As the user moves backand forth relative to the device 12, different colored combinations mayhelp guide his or her movement.

The image capture device 12 may also employ provide positional feedbackto the user by using the pupil tracking and iris focus measure digitalprocessing algorithms. A retina focus measure digital processingalgorithm can be used in place of, or in combination with, an iris focusmeasure digital processing algorithm.

In another fixation system 70 illustrated in FIGS. 8A-C, an interfaceprovides a set of central cross hairs 72 designating nominal positioning(X_(N), Y_(N), Z_(N)) for optimal alignment by the user relative to thedevice 12 and a second set of cross hairs 74 with a circle 76designating the user's present position (X₁, Y₁, Z₁). When the usermoves along the X- and Y-axes (left, right, up and down as shown inFIGS. 8A-C), the cross hairs 74 with the circle 76 correspondingly movealong the X- and Y-axes. When the user moves back and forth relative tothe device along the Z-axis the diameter of the circle 76 becomes largeras the user moves away from the nominal Z_(N) position and smaller asthe user moves towards the nominal Z_(N) position. When the circle 76disappears, the user is positioned at the nominal Z_(N) position.Furthermore, when the user sees only a single set of cross hairs, thesecond set of cross hairs 74 overlaps with the central cross hairs 72.Therefore, the image of FIG. 8A indicates that the user is misalignedalong the X-, Y-, and Z-axes. Meanwhile, the image of FIG. 8B indicatesthat the user is aligned along the X- and Y-axes, but misaligned alongthe Z-axis. When the interface shows the image of FIG. 8C, the user hasachieved the nominal position (X_(N), Y_(N), Z_(N)). Auditory feedbackmay be additionally employed with the fixation system 70, signaling theuser with appropriate tones and/or verbal instructions to move the userinto optimal alignment.

As shown in FIG. 9, embodiments of the present invention may include, incombination with the fixation system, a clock interface 80 that acceptsa pin number for further identification. When users look into thedevice, they begin by looking at a start position. They then enter theirpin number by fixating on the numbers or other symbols 82. The systemuses the pupil tracking to determine the trajectory of the differentpupil positions to identify each number or symbol 82 entered by theuser. Verification of each number or symbol 82 can be indicated throughaural tones and/or visible color changes, as illustrated by number 83 inFIG. 9.

In addition to the two-eye simultaneous iris/retina combination systemshown in FIG. 1, other configurations can be employed to combine irisand retina images. A left-eye only configuration employs iris and retinaimaging systems to capture images of the left eye only. Similarly, aright-eye only configuration employs iris and retina imaging systems tocapture images of the right eye only.

As shown in FIG. 3, a dual sensor, two-eye “flippable” system 20provides one iris imaging system 400 and one retina imaging system 500in an image capture device 22 that can be oriented to capture images ofboth the left and right eyes, in succession. In particular, the sameiris imaging system 400 and retina imaging system 500 are used tocapture images of the left and right eyes. Once images of one eye arecaptured, the user flips, or turns the device over, to capture images ofthe second eye. Flipping the device over maintains the correctorientation of the iris and retina imaging systems with respect to theeye. For example, the specific orientation shown in FIG. 2 permits thecapture of images from the right eye 4.

Similar to the iris imaging system 100 described previously, the irisimaging system 400 in FIG. 3 employs a camera sensor 410 which capturesimages of the illuminated iris through a dichroic beamsplitter 430.Similar to the retina imaging system 200 described previously, theretina imaging system 500 in FIG. 3 employs an illumination source 520that provides light that is guided through a LCDshutter/diffuser/micro-optics 525, a polarizing beamsplitter (PBS) 530,and an aspheric objective lens 540 to the retina. Furthermore, the imageof the retina then passes back through the aspheric objective lens 540and the PBS 530 to the camera sensor 510. The system 20, however,employs a dual fixation LED with orientation sensor 525, where one ofthe two LED's is activated according to the “flipped” orientation of thedevice 22. An orientation sensor senses the orientation of the device 22and correspondingly turns on the appropriate fixation LED.

Moreover, the system 20 as shown in FIG. 3 also uses a dual retina/irisillumination and retina illumination tracking configuration. In otherwords, the illumination source 520 provides illumination of the iris aswell as the retina. The retina illumination system in this embodiment issimilar to the illumination system for the retina in the two-eyesimultaneous system 10 shown in FIG. 1, where the element of the LEDarray is illuminated according to the pupil's position in space. Thecaptured iris image is used to track the position of the pupil in orderto identify the specific LED that should be used to provide thenecessary pinpoint illumination of the retina in the subsequent imagecapture. Here, however, both the retina illumination and irisillumination emanate through the retina imaging optics. The addressablelight source array 520 is used to create pulsed light for both iris andretina illumination. All elements in the array 520 are employed toilluminate the iris. Then, using the pupil tracking digital processingalgorithm and iris focus measure digital processing algorithm, selectedelements in the array are turned on to illuminate the retina. As theposition of the iris moves the appropriate elements in the array areselected for both the retina and iris. For the retina illumination, theillumination elements of the array imaged (to just before the retina)are smaller than the pupil of the iris. Advantageously, thisillumination configuration enables simplification of the packaging,minimizes reflections off the orbit of the eye for uniform irisillumination, and allows scanning of the retina for increased volume ofalignment. As described above, the addressable light source array can bebuilt in several different configurations, including, but not limitedto, the use of an LED array with light guides and diffuser and an LCDshutter, scanning micro-optics, and holographic elements, as indicatedby reference numeral 525.

In another embodiment illustrated in FIG. 4, a triple sensor, two-eyesequential system 30 employs one iris imaging system 600 and two retinaimaging systems 700 in device 32 to capture sequential, or successive,images of both the left and right eyes. The same iris imaging system 600is used to image the iris of both the left and right eyes, while tworetina imaging systems 700 with specific left and right orientations areused to image the left and right eyes, respectively. Unlike the two-eyeflippable system 20, the system 30 of FIG. 4 does not have to beflipped, or turned over, to capture images of the second eye. Thus, iteasier to reposition for capture of images from the second eye, becausethe same horizontal plane can be maintained. In addition, a dualfixation LED with orientation sensor does not have to be employed.Rather, a single fixation source 310 may be employed.

Similar to the iris imaging system 100 described previously, the irisimaging system 600 in FIG. 4 employs a camera sensor 610 which capturesimages of the illuminated iris through a dichroic beamsplitter 630.Similar to the retina imaging system 200 described previously, theretina imaging system 700 in FIG. 4 employs an illumination source 720that provides light that is guided through a LCDshutter/diffuser/micro-optics 725, a polarizing beamsplitter (PBS) 730,and an aspheric objective lens 740 to the retina. Furthermore, the imageof the retina then passes back through the aspheric objective lens 740and the PBS 730 to the camera sensor 710.

In yet another embodiment shown in FIG. 5, a single-sensor, two-eyesequential system 40 includes a single sensor 810 in device 42 tocapture both the retina and iris images by employing pulse separationand different wavelengths for the iris and retina. Wavelengthmultiplexing can be implemented with this embodiment, where a singleoptic with two surfaces with different coatings permits the capture ofdifferent images corresponding to particular wavelengths. For instance,λ₁=810 nm and λ₃=880 nm can be used to capture images of the iris, whileλ₂=850 nm and λ₄=910 nm can be used to capture images of the retina. Thetwo coated surfaces on the single optic permit sequential detection ofλ₁, λ₂, λ₃, and λ₄ and capture of alternating images of the iris andretina. In general, several optical systems can be used to get theimages of both eyes on a single detector array. Like the systemsdescribed above, the system 40 shown in FIG. 5 employs an LED array 820,a LCD shutter/micro-optics/diffuser 825, a polarizing beamsplitter (PBS)830, an aspheric objective lens 840, and a single fixation source 310.However, a compensating lens 865, an extra reflective mirror 860, and adichroic beamsplitter 870 are additionally used in order to form imagesof the retina and the iris on the same camera sensor 810. Thecompensation lens allows for proper image magnification for the captureof the iris by camera sensor 810. Moreover, similar to the system 20 ofFIG. 3, the system 40 uses a dual retina/iris illumination and retinaillumination tracking configuration. The advantage of this system isthat it uses a single sensor and fewer parts. However, the disadvantageis that the system runs at half the frame rate for iris and retina imagecapture being every other frame respectively. In other words, halvingthe frame rate yields half the number of images of the retina and theiris, so it may be more difficult to obtain adequate images. Inaddition, this particular configuration must also be flipped like thedual sensor, two-eye flippable configuration shown in FIG. 2.

When the retina illumination tracking system described above is usedwith symmetric iris/retina camera combinations to allow simultaneouscapture of both eyes, such as the two-eye simultaneous system 10 of FIG.1, one achieves automatic interpupillary adjustment without the need forany moving parts. Interpupillary distance measurement can be determined,providing an additional biometric. With information regarding positionalong the X- and Y-axes from the pupil tracking algorithm andinformation regarding position along the Z-axis from the focus measurealgorithm, the (X, Y, Z) position of each pupil can be used to calculatepupil separation. As described above, this particular biometric is usedto reduce database searching for iris matching, retina matching and irisretina fusion matching. Additional on axis illumination of the iris canalso enable bright pupil back reflection (“red eye”) that can enhancethe iris/retina tracking algorithms.

While all the embodiments above capture and process a combination ofiris and retina images, other embodiments of the present invention maycapture and process either images of the iris or the retina from botheyes of a subject. As described previously, biometrics based on datafrom both eyes are more accurate and robust than using biometrics thatinclude data from only the iris or only the retina from a single eye.Illustrating a corresponding exemplary embodiment, FIGS. 6A-D show adevice 50 adapted to simultaneously accommodate both eyes of a user,similar to a pair of binoculars, and capture images of both irises. Asshown in FIGS. 6A-D, the user employing the device 50 is able to seethrough the device 50 to view an object external to the device 50 as atarget. In particular, with the device positioned at the user's rightand left eyes, the user looks into the user windows 902 and throughopposing windows 904 to view the external object, or target, on theother side of the device 50. Unlike the embodiments describedpreviously, the device 50 may be employed without a fixationillumination source. The device 50 employs a fixation system where theexit pupil matches or is slightly larger than the entrance pupil of theeye for a given eye relief. Using a two-eyed simultaneous configurationaccommodating both eyes, an elliptical, or near elliptical, exit pupilis used to accommodate interpupillary distance. This maintains verticalalignment and allows vergence by the user to maintain horizontalalignment. The target may be an image which appears to be at a distance.Advantageously, this causes the brain to allow the eye to relax to itsunaccommodated state.

In particular, FIG. 6A shows the biometric device 50 with housing 900.In general, users begin by looking through the user windows 902 andbringing the device 50 closer to their eyes until they are able to usetheir vergence to visually fuse the exit pupils of the device 50. Thisapproach aligns most users to a given image plane with or withouteyeglasses. As shown in FIG. 6B, the device 50 also has opposing windows904 facing the opposing user windows 902. The opposing windows 904 notonly permit an image of the target on the other side of the device 50 tobe seen by the user, but the opposing windows also allow one to see theeyes of the user positioned at user windows 902. As a result, inaddition to operation of the device 50 directly by the user, the device50 also permits operation by another person who holds the device 50 atthe user's face and aligns it to the user's eyes from the other side.Thus, the device 50 allows an operator to assist a user during alignmentand operation of the device 50.

With its binocular-like shape, the device 50 helps to ensure properalignment about at least two axes of rotation in order to achieve abetter biometric. With reference to the X-, Y-, and Z-axes shown in FIG.6A, when users bring the device 50 to their face, they have to positionthe device 50 so that they can see through both user windows 902, thusensuring proper alignment about the Z-axis. Moreover, in order to lookinto the device 50 more easily, users naturally position the device 50so that the user windows 902 are approximately the same distance fromeach respective eye, which ensures proper alignment about the Y-axis. Asdescribed above, the exit pupils can then be elliptical, or slit-like,to minimize any misalignment about the X-axis.

Additionally, to obtain more precise alignment of the user's eyes, alinear horizontal diffraction grating, or equivalent “microlouver”technology, may be placed on user windows 902 in order to limit thefield of view of the user or operator and ensure proper alignment alongthe vertical Y-axis. A second vertical diffraction grating may also beemployed to also ensure proper alignment along the horizontal X-axis.The combination of horizontal and vertical gratings limits the field ofview vertically and horizontally. Moreover, a semitransparent target maybe placed behind the gratings for additional alignment indicators.

FIG. 6D illustrates an arrangement of components that may be employed bydevice 50 to capture images of the irises of both eyes positioned atuser windows 902. Two camera sensors 910 with filters 912 are positionedon opposite (right and left) sides in the interior of the device 50 tocapture respective images of the right and left eyes. The LEDs 920provide near infrared illumination to the iris of each eye. Theillumination is reflected from the irises back to the respectivebeamsplitters 930. The beamsplitters 930 may be “hot” mirrors whichredirect the near infrared light reflected from the irises to therespective camera sensors 910, but which allow visible light to passthrough to the operator windows 904 so that an operator can see theuser's eyes. White light illumination from the white light illuminationsources 950 may be employed to close down the pupil of the user toprovide better biometric data and to help illuminate the eye foralignment by an operator. As shown in FIGS. 6A and 6D, the area 922 ofnear infrared light cast by the LEDs 920 is smaller than area 952 ofwhite light cast by sources 950. With a smaller area 922, the amount ofnear infrared light reflected from the area outside the iris, such asthe user's cheeks, is minimized.

To facilitate the use of the device 50 by an individual who requirescorrective eyeglasses, the device 50 may accommodate the individual'seyeglasses 7, as illustrated in FIG. 6C. For instance, the individual'seyeglasses 7 may be combined with the device 50 beyond the beamsplitters930 but in a position where the eyeglasses can sufficiently correct theperson's vision in order to use the device 50 and view the externalobject. Accordingly, illumination and image capture are not affected bythe eyeglasses.

It is understood that a device similar to the device 50 illustrated inFIGS. 6A-D may be used to capture images of the retina from both eyes.Of course, another similar device may be employed to capture images ofboth the iris and the retina of both eyes, in a manner similar toembodiments described previously.

As described previously, various algorithms may be employed to processthe data captured by the multimodal ocular devices described herein. Forexample, the device 50 may employ the digital processing algorithms 1010illustrated in FIG. 10 to process and evaluate iris image data. Asdescribed with respect to various embodiments herein, such digitalalgorithms 1010 may include a pupil segmentation algorithm 1011 fordetermining a pupil image in the captured image, an iris segmentationalgorithm 1012 for determining an iris image in the captured image, aneyelid/eyelash segmentation algorithm 1013 for determining aneyelid/eyelash image in the captured image, and an algorithm 1014 formeasuring the focus on the iris. Moreover, device 50 may employ anauto-capture process which employs employ any of digital algorithms1010, in part, to evaluate captured images and obtain the best possibleimages for biometric identification, for example, as described withreference to FIGS. 21 and 22.

In some embodiments, a plurality of processing threads may process theplural sets of image data corresponding to the multiple modes of thedevices. For example, in a two-iris device, two iris processing threadsmay run in parallel. In a retina/iris device, an iris thread runs inparallel to a retina processing thread. In one particular embodiment,the controlling software waits for all processing threads to provide a“successfully acquired” signal. Preferably, each thread continuesprocessing until all threads have provided a “successfully acquired”signal. Therefore, with reference to FIG. 21, when a process reachesstep 1490 but other threads have not yet provided a “successfullyacquired” signal, then the process loops back to step 1420. On the otherhand, if multiple processing threads are in progress and a timeout issignaled in step 1480, the timeout halts all threads whereby each threadfinishes processing the current frame and all then return a“successfully acquired” or “failed to acquire” signal based on whetherthe number of image frames in the cache is greater than or less than anumber O.

The present invention may include time linking of image frames acrossdifferent threads. This may be achieved through the sequential indexingof frames as read from different cameras or though timing stamping imageframes using the PC clock.

As described above with reference to FIG. 1, the controller 15 may be aprogrammable processing device, such as an external conventionalcomputer networked with the device 12 or an on-board field programmablegate array (FPGA) or digital signal processor (DSP), that executessoftware, or stored instructions. Controllers 25, 35, and 45 shown inFIGS. 3, 4, and 5, respectively, may be similarly configured. Ingeneral, physical processors and/or machines employed by embodiments ofthe present invention for any processing or evaluation may include oneor more networked or non-networked general purpose computer systems,microprocessors, field programmable gate arrays (FPGA's), digital signalprocessors (DSP's), micro-controllers, and the like, programmedaccording to the teachings of the exemplary embodiments of the presentinvention, as is appreciated by those skilled in the computer andsoftware arts. The physical processors and/or machines may be externallynetworked with the image capture device, or may be integrated to residewithin the image capture device. Appropriate software can be readilyprepared by programmers of ordinary skill based on the teachings of theexemplary embodiments, as is appreciated by those skilled in thesoftware art. In addition, the devices and subsystems of the exemplaryembodiments can be implemented by the preparation ofapplication-specific integrated circuits or by interconnecting anappropriate network of conventional component circuits, as isappreciated by those skilled in the electrical art(s). Thus, theexemplary embodiments are not limited to any specific combination ofhardware circuitry and/or software.

Stored on any one or on a combination of computer readable media, theexemplary embodiments of the present invention may include software forcontrolling the devices and subsystems of the exemplary embodiments, fordriving the devices and subsystems of the exemplary embodiments, forenabling the devices and subsystems of the exemplary embodiments tointeract with a human user, and the like. Such software can include, butis not limited to, device drivers, firmware, operating systems,development tools, applications software, and the like. Such computerreadable media further can include the computer program product of anembodiment of the present inventions for performing all or a portion (ifprocessing is distributed) of the processing performed in implementingthe inventions. Computer code devices of the exemplary embodiments ofthe present inventions can include any suitable interpretable orexecutable code mechanism, including but not limited to scripts,interpretable programs, dynamic link libraries (DLLs), Java classes andapplets, complete executable programs, and the like. Moreover, parts ofthe processing of the exemplary embodiments of the present inventionscan be distributed for better performance, reliability, cost, and thelike.

Common forms of computer-readable media may include, for example, afloppy disk, a flexible disk, hard disk, magnetic tape, any othersuitable magnetic medium, a CD-ROM, CDRW, DVD, any other suitableoptical medium, punch cards, paper tape, optical mark sheets, any othersuitable physical medium with patterns of holes or other opticallyrecognizable indicia, a RAM, a PROM, an EPROM, a FLASH-EPROM, any othersuitable memory chip or cartridge, a carrier wave or any other suitablemedium from which a computer can read.

While the present invention has been described in connection with anumber of exemplary embodiments, and implementations, the presentinventions are not so limited, but rather cover various modifications,and equivalent arrangements, which fall within the purview ofprospective claims. For example, the positions of the iris camera andthe fixation illumination source in embodiments above may be switched bythe use of a “hot” mirror which reflects the iris image. Similarly, thepositions of the retina camera and the retina illumination may beswitched by illuminating the retina with P polarized light and imagingthe S polarized light.

As a further example, while embodiments may capture retina images fromboth eyes, only the best retinal image from both eyes may be retained toensure useable retinal biometric data. As a result, for a two-eyesimultaneous configuration, the embodiment produces data regarding thepupillary distance as well as biometric data from both irises and one ofthe two retina.

Moreover, although the exemplary embodiments discussed herein arecombination retina and iris imaging systems used for humanidentification, the multimodal ocular biometric system of the presentinvention is not limited to human identification and can be used foranimal identification.

What is claimed is:
 1. A method for segmenting a biometriccharacteristic from an eye image captured by a biometric device, the eyeimage comprising a pupil image, an iris image, and a sclera image, andthe method comprising the steps of: identifying an initial pupilboundary of the pupil image; applying a multiplicity of first searchwindows onto the eye image; identifying a refined pupil boundary of thepupil image based at least in part on the initial pupil boundary and amultiplicity of first search windows; identifying a center of the pupilimage based at least in part on the refined pupil boundary of the pupilimage; searching for a light-to-dark edge to the left of the pupilboundary to identify at least a first edge point between the scleraimage and the iris image; searching for a dark-to-light edge to theright of the pupil boundary to identify at least a second edge pointbetween the sclera image and the iris image; estimating an initialnon-occluded iris outer boundary of the iris image based at least inpart on the identified edge points and the center of the pupil image;applying a multiplicity of second search windows onto the eye image, themultiplicity of second search windows being based at least in part onthe initial non-occluded iris outer boundary for the iris image;identifying a set of points corresponding to peaks in an image intensitygradient in each second search window of the multiplicity of secondsearch windows; and defining a refined non-occluded iris outer boundaryfor the iris image based on the set of points.
 2. The method accordingto claim 1, wherein searching for the light-to-dark edge to the left ofthe pupil boundary and searching for the dark-to-light edge to the rightof the pupil boundary are based on two or more rectangular third searchwindows, each having a longitudinal axis that originates from the centerof the pupil image and extends downwardly from the center of the pupilat an angle from a horizontal line of the eye image.
 3. The methodaccording to claim 2, wherein each longitudinal axis extends downwardlyfrom the center of the pupil image at an angle in a range between 15degrees and 20 degrees from the horizontal line of the eye image.
 4. Themethod according to claim 1, further comprising identifying an annularregion corresponding to the iris image, the annular region being definedby the refined non-occluded outer boundary for the iris image and therefined pupil boundary.
 5. The method according to claim 1, wherein eachsecond search window of the multiplicity of second search windows is arectangular search window having a longitudinal axis extendingperpendicularly to the initial non-occluded iris outer boundary.
 6. Themethod according to claim 1, wherein each first search window of themultiplicity of first search windows is a rectangular search windowhaving a longitudinal axis perpendicular to the initial pupil boundary.7. The method according to claim 1, further comprising assessing anamount of occlusion of the iris image.
 8. The method according to claim1, wherein the initial non-occluded iris outer boundary and the refinednon-occluded iris outer boundary are estimated without regard to theassociated eyelid and eyelashes.
 9. The method according to claim 1,wherein identifying the initial pupil boundary of the pupil imagecomprises defining a multiplicity of third search windows in the eyeimage.
 10. The method according to claim 9, wherein each third searchwindow in the multiplicity of third search windows is rectangular. 11.The method according to claim 10, wherein each third search window inthe multiplicity of third search windows extends horizontally across theeye image.
 12. The method according to claim 9, wherein identifying theinitial pupil boundary of the pupil image further comprises: identifyinga second set of points corresponding to peaks in an image intensitygradient in each third search window in the multiplicity of third searchwindows; determining, from the second set of points, segments accordingto image intensity transitions, each segment having a center point;determining a position for the center point of each segment; groupingsets of the segments into a first set of clusters according to ananalysis of the positions of the center points for the segments;selecting, from the first set of clusters, a second set of clusterscorresponding to the pupil image; and fitting a model template to thepoints for each set of segments corresponding to the second set ofclusters, wherein the fitted model template is the initial pupilboundary of the pupil image.
 13. The method according to claim 12,wherein the image intensity transitions comprise a light-to-darktransition on a first side of the pupil image and a dark-to-lighttransition on a second side of the pupil image.
 14. The method accordingto claim 12, wherein selecting the second set of clusters comprisesselecting, from the first set of clusters, corresponding to a shapefeature.
 15. The method according to claim 14, wherein selecting thesecond set of clusters further comprises applying a weighting score toeach cluster in the first set of clusters according to pupil-relatedscoring criteria and sorting the first set of clusters according to theweighting scores.
 16. The method according to claim 1, furthercomprising identifying a second set of points corresponding to peaks inan image intensity gradient in each first search window of themultiplicity of first search windows, and fitting a model template tothe second set of points, the fitted model template corresponding to therefined pupil boundary of the pupil image.
 17. The method according toclaim 16, wherein the model template is a circle.
 18. The methodaccording claim 1, wherein estimating the initial non-occluded irisouter boundary of the iris image comprises fitting a model template toidentified edge points.
 19. The method according to claim 1, whereindefining the refined non-occluded iris outer boundary comprises fittinga model template to the set of points corresponding to the peaks in theimage intensity gradient in each of the multiplicity of second searchwindows.
 20. The method according to claim 19, wherein the modeltemplate is a circle.